Conventional image capture or display devices are prone to various forms of optical anomalies. These anomalies are inherent to the non-ideal behavior of the various optical elements and to accuracy and tolerance of assembly. Various components (sensors, displays, lens, prisms, mirrors, light source), optical or otherwise, and their orientation may introduce their own specific optical anomalies, such as distortion, tilt, lateral chromatic aberration, luminance or chrominance non-uniformity. The term optical aberration is generally used to refer to any effect that leads to an non-ideal image formation.
Optical aberrations include diffraction effects (due to the wave nature of light), chromatic aberrations (caused by optical dispersion, or the differences in refraction in different wavelengths of light), and monochromatic aberrations (of which spherical aberration, coma, and astigmatism are concerned with failures of a point object to form a point image, and field curvature and distortion are concerned with the failure of finite objects perpendicular to the principal axis to form a well-focused plane image). One can also loosely group optical aberrations into two types, ones that affect image quality and ones that affect image shape. The former types degrade the sharpness of an image, that is, the image appears blurred, and/or out of focus, and/or has color fringes. Aberrations in this category include spherical aberrations, astigmatism, coma, field of curvature and axial chromatic aberration. The latter types of aberrations affect the shape of the image that in part may be induced by the former aberration or by the correction and optimization of the former aberration. In this case, points in the object plane are shifted or distorted in comparison with an ideal mapping in the image plane. In an ideal mapping an object in the image plane will appear as it does in the object plane, with possibly a uniform scale change. For example, an image may appear curved near the edges or appear rotated. Aberrations in this second category include distortions (e.g. pincushion/barrel effects) and lateral chromatic aberrations.
With the exception of chromatic aberrations, all other optical aberrations are present in monochromatic (i.e. single color) light. Chromatic aberration appears when dealing with polychromatic light (many colors). In short, the index of refraction is wavelength dependent, which means that the red, green and blue components bend differently at an optical interface. This leads to axial (longitudinal) and/or lateral chromatic aberration effects. In axial chromatic aberration, the three components are brought to focus on different planes in the image space, which gives a color blurring effect. In other words, axial chromatic aberration arises due to the focal length varying with wavelength (color). In lateral chromatic aberration, color components from a single point are brought to focus to different points on the same image plane. This has the effect of magnifying the three colors differently and can be visually seen as ‘color fringing’. Thus lateral chromatic aberration can be seen as an effect due to magnification varying with wavelength. The three colors can also mismatch due to non-optical effects. In a three-color display system, if the displays are not correctly aligned, color defects will be seen. The term color non-convergence is used to refer to color mismatch effects, whether optical (as in chromatic aberrations) or not. Further discussion on optical aberrations can be found in conventional optics textbooks, such as Robert Guenther's Modern Optics, published by John Wiley & Sons, 1990, hereby incorporated by reference.
Another important optical anomaly in conventional capture/display devices is luminance non-uniformity. Luminance non-uniformity leads to varying brightness across an image. Common causes include a varying (in brightness) light source, varying optical path across the image plane, non-uniform sensor response and irregularities in panels (e.g. LCD, LCOS, etc.). Both large-scale and small-scale non-uniformities can be present. In a three-color system, brightness variation can be different for each color, leading to chrominance non-uniformity.
There have been a number of prior art attempts to correct aberrations that affect the shape of the image, without introducing blur, namely correction of distortion, lateral chromatic aberration and luminance or chrominance (brightness) non-uniformity. Generally, such prior art attempts are geared towards one specific type of anomaly.
For example, lateral chromatic aberration is commonly corrected (or minimized) using special optical elements, often consisting of prism/lens combinations and/or special material coatings such as that disclosed in U.S. Pat. No. 4,943,155 to Cross, U.S. Pat. No. 5,086,338 to Usui, U.S. Pat. No. 5,499,139 to Chen et al., U.S. Pat. No. 6,023,375 to Kreitzer, U.S. Pat. No. 6,111,701 to Brown, U.S. Pat. No. 6,144,498 to Bryars et al., and U.S. Pat. No. 6,172,815 to Hashizume et al. However, the physical implementation of the solutions disclosed in these references are expensive and bulky. Further, the specialized nature of these designs necessarily restrict them to specific types of applications. Typically, these methods are aimed at display/projection systems or head up/mounted display systems.
A number of electronic solutions have also been presented such as those disclosed in U.S. Pat. No. 5,838,396 to Shiota et al., U.S. Pat. No. 5,870,505 to Munib et al., U.S. Pat. No. 6,288,756 to Shiota et al., U.S. Pat. No. 5,200,815 to Tsujihara et al., U.S. Pat. No. 5,369,450 to Haseltine et al, U.S. Pat. No. 5,889,625 to Chen et al., and U.S. Pat. No. 6,323,934 to Enomoto et al. All of these approaches rely on some type of image “warping”. A discussion of image warping can be found in George Wolberg,Digial Image Warping, IEEE Computer Society Press, 1988, hereby incorporated by reference.
The warping data (i.e. data which describes how the image is to be transformed) may be used to adjust the digital image (e.g. as in U.S. Pat. No. 5,369,450 to Haseltine et al. and U.S. Pat. No. 5,889,625 to Chen et al.) or to adjust the operation of the electronics that display/project the image (e.g. as in U.S. Pat. No. 5,200,815). These electronic solutions concentrate on specific anomalies, such as luminance correction (e.g. U.S. Pat No. 5,838,396 to Shiota et al., U.S. Pat. No. 5,870,505 to Woeber et al, U.S. Pat. No. 6,288,756 to Shiota et al.), distortion and chromatic aberration (e.g. U.S. Pat. No. 5,200,815 to Tsujihara et al., U.S. Pat. No. 5,369,450 to Haseltine et al, U.S. Pat. No. 5,889,625 to Chen et al., and U.S. Pat. No. 6,323,934 to Enomoto et al.) or specific types of systems, such as head-mounted displays. Those solutions that do correct for all three anomalies (e.g. U.S. Pat. No. 6,323,934 to Enomoto et al.) are not real-time in nature. Other limitations of prior art electronic solutions are that they do not allow for application specific “correction” (i.e. correction which does not correspond to an optical anomaly correction) and/or they do not provide for dynamic anomaly correction. For example, it can be desirable in certain video applications to combine anomaly correction (brightness non-uniformity as well as pincushion distortion) with a keystone correction (caused by off-axis projection) and curvature correction for planar/curved surfaces.